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In a method of detecting 
covered and uncovered parts in an 
image to be interpolated between 
neighboring previous (I) and 
next (III) input images, backward 
motion vectors from the next 
input image to the previous input 
image, and having corresponding 
backward estimation errors, and 
forward motion vectors from 
the previous input image to the 
next input image, and having 
corresponding forward estimation 
errors, are determined, uncovered 
and covered parts are detected 
in the neighboring previous and 
next input images, respectively, in 
the thus detected uncovered parts, 




uncovered parts in the image to 
be interpolated are detected by 

determining second backward estimation errors by comparing both neighboring previous and next input images when partially shifted over 
the backward motion vectors to a temporal location of said image to be interpolated, and by comparing the second backward estimation 
errors to a threshold, and in the thus detected covered parts, covered parts in the image to be interpolated are detected by determining 
second forward estimation errors by comparing both neighboring previous and next input images when partially shifted over the forward 
motion vectors to a temporal location of the image to be interpolated, and by comparing the second forward estimation errors to the 
threshold. 
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1 

Motion vector estimation and detection of covered/uncovered image parts. 



The invention relates to motion vector estimation and detection of 
covered/uncovered image parts. The invention also relates to motion-compensated 
interpolation, and to a display apparatus comprising means for carrying out such a motion- 
compensated interpolation. 



The paper "Motion estimation method using the spatio-temporal 
characteristics of moving objects", presented at the International Workshop on HDTV, Los 
Angeles, October 1996, by H. Sonehara et al. f discloses a motion vector detection method 

10 which takes account of the spatio-temporal continuity of motion vectors and the local 

characteristics of pictures. Motion vector trajectories are traced to assess the spatio-temporal 
continuity of motion vectors. To remove erroneous motion vectors detected in the 
covered/uncovered background of the contour area of moving objects, motion vectors are 
estimated by means of a detection-area hierarchy and the correlation of luminance signals. It 

15 is necessary to identify, in two sequential fields, the area (covered and uncovered background 
area) in which the picture correlation becomes indeterminate. For this, a detection block is 
set in the current field and vectors are detected in relation to the next field. Then a detection 
block is set in the next field and vectors are detected in relation to the current field. In short, 
motion vectors are detected two-way. 

20 EP-B-0,535,066 discloses a method of assigning the appropriate one (if 

any) of a plurality of possible motion vectors to pixels of a video signal consisting of a series 
of pictures, the method comprising the steps of: a backwards assignment operation in which 
a first input picture is shifted by each of a plurality of possible vectors and compared to a 
second, succeeding picture to determine assignment errors for each vector at each pixel; a 

25 forward assignment operation in which the second picture is shifted by each of a plurality of 
possible vectors and compared to the first picture to determine assignment errors for each 
vector at each pixel; and using for each pixel the assignment errors determined in both 
forward assignment and backwards assignment operations to determine whether any of the 
vectors is appropriate for that pixel. The method may further comprise comparing the 



WO 99/22520 PCT7IB98/01594 

2 

assignment errors for the forward and backwards assignment operations for a pixel to 
determine whether the pixel may relate to revealed or obscured background. This combined 
vector assignment flags regions of both revealed and obscured background as 'no match' so 
that a fallback interpolation technique can be used. This fallback interpolation technique has 
5 not been described. In an alternative method, vectors from the instant of an input picture are 
converted to an output picture at a different instant in time by projecting the input vectors 
forward in time (allowing for their motion). Several problems are inherent in projecting 

vectors forward in time. While these problems are not insuperable the technique of combined J 
forward and backward assignment is simpler and therefore more attractive. 

10 

i 

It is, inter alia, an object of the invention to provide a reliable motion 
detection / motion-compensated interpolation. To this end, a first aspect of the invention 
provides a motion vector estimation method and device as defined in claims 1 and 5. A 

15 second aspect of the invention provides an interpolation method as defined in claim 4. A 
third aspect of the invention provides a display apparatus as defined in claim 6. A fourth 
aspect of the invention provides a method of and device for detecting covered and uncovered 
parts in an image to be interpolated as defined in claim 7 and 9. A fifth aspect of the 
invention provides a motion vector estimation device as defined by Claim 10. Advantageous 

20 embodiments are defined in the dependent claims. 

A primary aspect of the invention provides a method of detecting covered 
and uncovered parts in an image to be interpolated between neighboring previous and next 
input images, in which backward motion vectors from the next input image to the previous 
input image, and having corresponding backward estimation errors, and forward motion 

25 vectors from the previous input image to the next input image, and having corresponding 
forward estimation errors, are determined, uncovered and covered parts are detected in the 
neighboring previous and next input images, respectively, in the thus detected uncovered 
parts, uncovered parts in the image to be interpolated are detected by determining second 
backward estimation errors by comparing both neighboring previous and next input images 

30 when partially shifted over the backward motion vectors to a temporal location of said image 
to be interpolated, and by comparing the second backward estimation errors to a threshold, 
and in the thus detected covered parts, covered parts in the image to be interpolated are 
detected by determining second forward estimation errors by comparing both neighboring 
previous and next input images when partially shifted over the forward motion vectors to a 
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temporal location of the image to be interpolated, and by comparing the second forward 
estimation errors to the threshold. 

Another aspect of the invention provides a method of estimating motion 
vectors for an image to be interpolated between neighboring previous and next input images, 

5 in which motion from the next input image to the previous input image is detected to obtain 
backward motion vectors having corresponding backward estimation errors, motion from the 
previous input image to the next input image is detected to obtain forward motion vectors 
having corresponding forward estimation errors, uncovered and covered parts are detected in 
the image to be interpolated, motion vectors are furnished for the uncovered parts in 

10 dependence upon the forward motion vectors taken from locations wholly and/or partly 
shifted over the backward motion vector, and motion vectors for the covered pans are 
furnished in dependence upon the backward motion vectors taken from locations wholly 
and/or partly shifted over the forward motion vectors. 

These and other aspects of the invention will be apparent from and 

15 elucidated with reference to the embodiments described hereinafter. 

In the drawings: 

Fig. 1 illustrates covering and uncovering background situations; 
20 Fig. 2 illustrates where in the interpolated image covering and uncovering 

occurs; and 

Fig. 3 shows a preferred embodiment of the current invention. 

25 A primary aspect of the present invention aims at a strong reduction, if 

not an elimination, of halo effects for systems that have access to both neighboring fields by 
analyzing in which of the two fields the information, that has to be interpolated, is present 
(i.e. not covered) using two motion vectors per location. This information is used to 
interpolate the motion compensated picture from both neighboring images ignoring pixels 

30 from either of the two images that were either covered, or uncovered. The use of two motion 
estimators in parallel is considered to be realistic, even in consumer products, since earlier 
work [7, 8] has reduced the cost of a motion estimator to an affordable level. Nevertheless, 
an alternative will be discussed in which the additional calculations are only required in areas 
where covering or uncovering is expected, i.e. near picture parts in which discontinuities 
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(contours) in the vector field occur. 

Basic requirement for the current algorithm is the availability of two 

motion vectors for every location. The first vector D, is the one calculated by shifting a 
previous field to optimally match a current field, whereas the second is obtained by shifting 

5 the data from the current field to optimally match a previous field. The required 

segmentation of the image in "normal, "covering", and "uncovering" regions will be dealt 
with later. First, we shall elucidate the motion vector calculation using the full-search 
block-matching motion estimator to calculate the motion vectors. This algorithm is not a very 
cost effective approach to the motion estimation problem, but the more effective ones are 

10 based on the same principle. 

In block-matching motion estimation algorithms, a displacement vector is 
assigned to the center X = (Xx, Xy) T of a block of pixels B(X) in the current field n by 
searching a similar block within a search area SA(X), also centered at X, but in the previous 
15 field n-1. The similar block has a center, which is shifted with respect to X over the 
displacement vector Z)(X, n). To find D(X, n), a number of candidate vectors C are 
evaluated applying an error measure (C, X, n) to quantify block similarity. More formally, 
CS 1 " 2 * is defined as the set of candidates C, describing all possible (usually integer) 
displacements with respect to X within the search area SA(X) in the previous image: 

20 

CS™* = {C | -N <. C x _< N, -M <. C y <_ M} (1) 

where N and M are constants limiting SA(X). Furthermore, a block B(X) centered at X and 
of size X by Y consisting of pixel positions x = (x,y) T in the present field n, is now 
25 considered: 

B(X) = {x | X x - X/2 <. x < X x + X/2 A X y - Y/2 y < Xy 4- Y/2} (2) 



The displacement vector D(X, n) resulting from the block-matching 
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process, is a candidate vector C which yields the minimum value of an error function 
(C, X, n): 

J5(X, n) G {C € CS™ | e(C,X, n) <. e(7,X, n) V7 e CS™} (3) 

5 

If, which is the common case, the vector D(x , n) with the smallest 
matching error is assigned to all pixel positions x in the block B(X): 



Vjc e B(X): D(x, n) = D(X t n) (4) 

10 

rather than to the center pixel only, a large reduction of computations is achieved. 

As an implication, consecutive blocks B(X) are not overlapping. The 

error value for a given candidate vector C b is a function of the luminance values of the 
pixels in the current block and those of the shifted block from a previous field, summed over 
15 the block B(X). A common choice, which we will use, is the sum of the absolute 
differences: 



e>(C f £n)= £ *-D - F(x + C,, n)\ (5) 

fc SCO 

20 although a mean square error or a normalized cross correlation function are sometimes 
proposed. 

The difference between the first and the second motion estimator, using 
candidates C b and C, respectively and resulting in motion vectors D b and D fy is that we 
propose in the method according to the invention, lies in the error criterion of (5). According 
25 to (5), the current field is shifted to optimally match the previous field (backward 

prediction). The second estimator that we propose is identical to the first, apart from (5) 
which is replaced by: 
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€, (C, X, n) = £ |F(Jc, n) - F(f - C f n-l)\ (6) 



This implies that D f is optimally suited for forward prediction. With two 
estimators defined above, a high match error of the best matching candidate means that it is: 
5 . • difficult to predict the current field with the previous, for the forward predicting 

estimator applying (6), or that it is: 
• difficult to predict the previous field with the present, for the backwards 

predicting estimator applying (5). 

10 The first situation occurs in case of uncovering background, the other is 

typical for a covering of the background. The relative performance RP of the two motion 
estimators is now proposed to decide upon covering and uncovering: 

RP- (c »~^ (7) 

(e b + e,) + 6 

and: 

J covering, (RP±Th x ) 

15 decision = s /DD , t.n \°) 

I uncovering, (RPz-TnJ 



where 6 is introduced to prevent decisions of covering/uncovering in case of small match 
errors. In the experiments 5 was set to approximately 1 percent of the maximum error. 

20 However, preferably the following decision is taken: 



. _ Uncovering, (e^€, * Th^Th^ 
decision = \ , ' , ~- . (y/oaj 

\ covering, (e f z-e b * l/Th x +Th 2 ) 

where Th 2 is introduced to prevent decisions of covering/uncovering in case of small match 
errors. In the experiments Th, was set to approximately 3 and Th 2 to 20. 



25 



When a block is detected as belonging to either the covering or the 
uncovering region, the 8 blocks surronding it are also assigned to that region, in order to 
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improve the spatial consistency. 

Fig. 1 illustrates the covering and uncovering background situations. A 
moving object, a circle, is shown moving (with motion vector D) from left to right. N 
5 indicates the position in field n, while N + l indicates the position in field n+l. Region I is 
uncovered in field n + l, while for region III covering occurs. Region II can be interpolated 
from both fields. 

Although this brings some clarity about areas where covering and 
uncovering appears, the precision of the method is not optimally suited for the image(s) to be 
10 interpolated. In that image only part of region I is uncovered and only part of region III 
becomes covered. The fraction depends on the temporal position of the interpolated image 
relative to the neighboring original images. If the interpolated image has to be valid at the 
temporal instance exactly in between the previous and next original, than the covering and 
uncovering areas are half the size that we detected. 

15 

Fig. 2 illustrates this case into more detail, and shows, particularly, 
regions in which covering and uncovering occurs, and where in the interpolated image which 
motion vector can be safely used, and from which, next or previous, image data should be 
used in the interpolation process. Region la can be interpolated from the next image using 

20 vector D v Region lb can be interpolated from the next and the previous image using vector 

D b . Region Ilia can be interpolated from the next and the previous image using vector D f . 

Region Illb can be interpolated from the next image using vector D v Two areas still exist, 
i.e. Ia and Illb, for which no suitable motion vector has been identified. Hereinafter, a 
practical solution for all regions will be introduced. 



25 



30 



So far, we distinguished areas in which covering and uncovering occurs 
between two successive images. However, at the temporal instance where the new image is 
interpolated we need to refine our earlier recognition of covering and uncovering based on e b 
and € f . To this end, we define e b ' and e f ' according to: 



* b 0 b , X, it) 



' £ (F(x-aD bJ n-1) - F(x * (l-a)D,, n)) 4 W B , 



(9) 
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and 

e l f 0 r I n) = ( £ [F&-*D f n-1) - F(f + (l-a)D f n^V, (10) 

where N B1 stands for the number of samples in block B,(X), Instead of a fourth power, a 

second power can also be used. The block B,(X) is not necessarily identical to B(X). A 
5 smaller block size will mostly be preferred, as this allows a more detailed region 

categorization. This error measure has been chosen since it has proven to be the most 

effective, in most cases. 

While we are within region I, it is concluded if e b * <L Th, that we are in 

region lb, and in region la otherwise. The variable Th, here, is an experimentally 
10 determined, small, threshold value. In words: we assume to be in region lb when D b can be 

considered valid. This occurs when, at the temporal position of the image to be interpolated, 

the two luminance values obtained according to Z) b , i.e. the one from the delayed field and 

the one from the current field, can be matched with a small error. We assume to be in region 

la when D b cannot be considered valid. 
15 Similarly, while we are within region III, we conclude that if ef <. Th, 

that we are in region Ilia, and in region Illb otherwise. In words: we assume to be in region 

Ilia when D f can be considered valid. This occurs when, at the temporal position of the 
image to be interpolated, the luminance values obtained according to D f can be matched with 
a small error. We assume to be in region Illb when D { cannot be considered valid. 

20 

It can be concluded that for any image to be interpolated between the 
previous and the current field, it seems best to interpolate the result from the previous field 
only, or mainly, in case of 'real covering' of the background (region Illb), whereas in case 
of 'real uncovering' of the background (region la), the motion compensated data from the 
25 next field is preferred in the interpolation process. In all other cases, the motion compensated 
interpolator can use the data from both fields. 



So, where a prior art motion compensated average would yield an 
intermediate picture according to: 
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F.(x, n) = 1/2 (F(x - aD(x, n), n-1) + F(f + (l-a)D(J, w), «)) (11) 

where a determines the temporal position of the interpolated image, we now propose to use 
different interpolation strategies for the various regions recognized according to the previous 
section. For example: 

5 

n) = 

F(x - aZ5 ? (jc, n), n - 1) , region IU b 

F(Jc + (l -a)2> ? (£, n), n) » region 7, 

A(F(Jc - aD/x, n), n - 1) + F(jc + (1 - a)DJ& w), n)) , region UI a 



±(F(x-aD b (x, n), +F(J?+(1 «), n)) , region 7 fc 
|(F(Jc - aZ>/x, n) f n-l) + F(x + (1 - a)D b (x, n), n)) , otherwise 



Rather then switching between the various interpolation methods in (12), a 
soft switch could be thought of, i.e. a fading between the five alternatives. We will, 
10 however, not further elaborate this thought here. 

In (12), and also in Fig. 2, vectors Z) ? occur, which up to now have no 
meaning. They are required for those parts of the interpolated image where there is either 
covering or uncovering. Therefore it is known that either the previous or the next image 
should be solely used in the interpolation, but no valid vector is available to shift the 
15 corresponding background that either appears or disappears. 

In order to solve this problem, we here propose to search for the nearest 
vector describing the background velocity. If we are in the uncovering region I, this nearest 

vector is obtained from the backward predicting motion estimator, generating D bi at a 

location shifted over the negative value of the forward predicting motion vector D f , 

20 multiplied by a, the temporal position of the image to be interpolated. If we are in the 
covering region III, this nearest vector is obtained from the forward predicting motion 
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estimator, generating D ft at a location shifted over the negative value of the backward 
predicting motion vector D b , multiplied by (1— a). So, more formally: 



D f (x - <x£>., n) , (region I, i.e. 'real uncovering) 
D (x /t) ~ i ( 1 ^ ) 

7 ' p 6 (x + (1 - a)D f , n) , (region III^ Le. 'real covering') 

5 

An alternative, prior art, motion compensated interpolation was described 
in [1]. It uses an order statistical filter to make the upconversion more robust with regard to 
erroneous motion vectors. A very basic version was described in [6], and applied in the 
Natural Motion IC [9]. This version uses: 

10 

F t (f, n) = med {F(x - aD(x, n), Av, F(x + (1 - a)D(x, n) } n)) (14) 
with 

Av = 1 n) + F(x, n-1)) (15) 
2 

and 

{a , (bzazc V csazb) 
b , (azbzc V czbsa) (16) 
c , otherwise 



In regions lb and Ilia a safer version of the interpolation according to the 
current invention would use a similar approach. As can be seen from the equations (18) 
below, motion compensated weighted averaging is proposed for 'uncomplicated' areas. In 
20 region lb and Ilia, i.e. the areas in which there is no actual covering in the image to be 
interpolated, but yet near a complicated region, the robust interpolation according to (14) is 
proposed. 

In the 'real' covering and uncovering areas, la and Hlb, we would like as 
well to propose a robust interpolation method. Thus we again make use of a median filter. 
25 As an input to the median filter, our best guess is the pixel from the previous/current frame 
(depending on covering/uncovering), shifted over D ? . Another useful 'candidate pixel' is the 
pixel selected when shifting over the vector that would be valid for the neighboring a/b area 
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(la, in case we are in the uncovering region, IHb otherwise). The third pixel that we propose 
here is the one from the same field shifted over D™, where: 



D 17 {x f n ) I ^ ^ + g w j ^ (region IH b , i.e. 'real covering') 



5 



or preferably, 



2 

D f (x - (a+ -a)^, *) , {region I a , i.e. 'real uncovering') 

2 - (17a) 
D b (x + ((l-a)+-a)D / , n) , (region III b , i.e. 'real covering') 



The thought behind this choice is that D ? = D ?? in most cases where D ? is 
reliable. In those cases therefore our best guess is actually selected as the output pixel. If D ? 
10 and D ?? are not identical, it is likely that confusion exists between region a and b, and the 
chance increases that the second 'candidate pixel' is chosen. Eventually the sequence would 
be processed according to: 



15 



Ffc n) = 



med {F(x - aZ> ? (x, n), n-l), F w ^ F(x - aD^x, n\ n-l)) , region III b 

med {F(x + (l-a)Z5 ? (x, n), n), F(x + (l-a)D ?? (x, n), n)) , region I Q 

med [F(x - aD/x, n), n-l), Av, F(x + (l-a)D/x, n), n)) , region III a 

med [F(x - aD b (x, n), n-l), Av, F(x + (l-a)D b (x, n), n)) , region I b 

^ (F(x - aD^Jc, n), n-l) + F(x + (l-a)D b (x, n), n)) , otherwise 



(18) 



where: 

F ma = \ ( F (* ~ ai5 /* »)• ""D + F (* + »). ")) (19) 
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and 

Flb = 2 ' F(f " n) ' n ' l) + F(J? + (I " a) ^' n) ' W) ) (20) 

In order to provide a safer interpolation strategy for the most critical regions (la and Illb) an 
other solution has been tested. 



For region I 3 : 

n) = med 5 (F(x + (1 - a)D ? (f, n\ n\ 
F(x + (1 - a)D v (x, n), n\ 
Av, 

F(x - aD b (x, n), n - 1), 
F(x + (1 - a)D,(x, n), n)) 



(18a) 



10 

and for region III b : 

Fff, n) = mctf 5 (F(x + - aD ? (f, n), n - 1), 
F(x - a£) ?7 (x, n), n - 1), 

Av, (18b) 
F(x - aZ)^*, /z), 7Z - 1), 
F(Jc 4- (1 - a)5/jf, n), n)) 



This sophistication experimentally proved to significantly increase the 
interpolation quality. A preferred embodiment of the current invention is shown in the block 

15 diagram of Fig. 3. An input video signal I is applied to a dual estimator DE having two 
motion vector estimators ME1, ME2. In the dual estimator DE, the input video signal I is 
applied to a field delay FM1, a first input of the first motion vector estimator ME1, and to a 
first input of the second motion vector estimator ME2 thru a shifter S2. An output signal of 
the field delay FM1 is applied to a second input of the second motion estimator ME2, and to 

20 a second input of the first motion vector estimator ME1 thru a shifter SI. The first motion 
vector estimator ME1 supplies the vector Df, while the second motion vector estimator ME2 
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supplies the vector Db. 

The input video signal I is also applied to a motion-compensated 
interpolator MCI. In the motion-compensated interpolator MCI, the input video signal I is 
applied to a field delay FM2, and to a shifter S3. An output of the field delay FM2 is 

5 applied to a shifter S4. The shifters S3, S4 are controlled by the motion vectors Db, Df 
received from the dual estimator DE. Outputs of the shifters S3, S4 are applied to a median 
circuit med and to an average circuit Av. Outputs of the median circuit med and the average 
circuit Av are applied to a multiplexer MUX which supplies the output signal O to a display 
device CRT for displaying the output signal at, for example, a 100 Hz field rate. The motion 

10 vectors Db, Df, their errors 6b, ef, and threshold values ml, m2 are applied to a region 
detector RD which furnishes a control signal to the multiplexer MUX. 

A primary aspect of the invention can be summarized as follows. In 
pictures resulting from motion compensated field rate convenors an artifact is visible at the 

15 boundary of moving objects where either covering or uncovering of backgrounds occurs. It is 
the object of the present invention to eliminate this local artifact. To this end, areas where 
covering and uncovering occur are recognized based on the relative prediction quality of two 
motion vectors. The two motion vectors are estimated between the same field pair, but differ 
in that the first is calculated to optimally predict the previous field from the next, whereas 

20 the second tries to predict the next field from the previous. An upconversion method using 
the two motion vectors is also described in this disclosure. 

In a preferred interpolation method, two motion vectors, one predicting 
the next image from the previous image and the other predicting the previous image from the 
next, are taken from both sides of a discontinuity in the vector field, and both applied to 

25 predict the next image from the previous and to predict the previous image from the next, 
and a choice between the vectors is made in dependence on the relative suitability for the 
tested prediction. 

A preferred embodiment can be summarized as follows. An image 
between neighboring images in a sequence, is interpolated by applying two motion vectors 

30 V b and V p one optimally suited for backwards predicting and the other for forwards 

prediction. The embodiment comprises: 

• means to shift data from the next image over a fraction of V b , 
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• means to shift data from the previous image over a complementary fraction of V f , 

• means to shift data from the previous image over a fraction of V fl 

• means to shift data from the next image over a fraction of V h , 

• means to combine the shifted data, and possibly other (unshifted) data in dependence on 
5 segmentation information indicating covering/uncovering in the interpolated image, in order 

to form the interpolated image characterized in that in case of covering/uncovering, at least 
one of the vectors that are used in the combination process are taken from a spatial position 
that differs from the spatial position for which it was calculated. Preferably, the position 
from which the vector is taken, is shifted in the opposite direction in which the vector is 
10 pointing with regard to the position for which it was calculated. Advantageously, in case of 

covering at least aV f is used shifted over -aV b , while in case of uncovering at least aV b is 
used shifted over -aV f . 

It should be noted that the above-mentioned embodiments illustrate rather 
15 than limit the invention, and that those skilled in the art will be able to design many 
alternative embodiments without departing from the scope of the appended claims. In the 
claims, any reference signs placed between parentheses shall not be construed as limiting the 
claim. The word "comprising" does not exclude the presence of other elements or steps than 
those listed in a claim. The invention can be implemented by means of hardware comprising 
20 several distinct elements, and by means of a suitably programmed computer. In a device 
claim reciting a plurality of different means, several of these means may be embodied by a 
one and the same item of hardware. In an alternative embodiment, the covered and 
uncovered parts are not detected by comparing backward and forward estimation errors, but 
by analyzing direction and magnitude of motion vectors around contours. 
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Claims: 



1. A method of detecting covered and uncovered parts in an image (II) to be 
interpolated between neighboring previous (I) and next (III) input images, the method 
comprising the steps of: 

detecting motion from said next input image (III) to said previous input 
5 image (I), resulting in backward motion vectors (D b ) having corresponding backward 
estimation errors (e b ); 

detecting motion from said previous input image (I) to said next input 
image (III), resulting in forward motion vectors (D f ) having corresponding forward 
estimation errors (e r ); 

10 detecting uncovered (I) and covered (III) parts in said neighboring 

previous (I) and next (III) input images, respectively; 

said method being characterized by the further steps of: 
detecting in said thus detected uncovered parts (I), uncovered pans (la) in 
said image (II) to be interpolated by determining second backward estimation errors by 
15 comparing both neighboring previous (I) and next (III) input images when partially shifted 
over said backward motion vectors (D b ) to a temporal location of said image (II) to be 
interpolated, and by comparing said second backward estimation errors to a threshold; and 
detecting in said thus detected covered pans (III), covered pans (IHb) in 
said image (II) to be interpolated by determining second forward estimation errors by 
20 comparing both neighboring previous (I) and next (III) input images when partially shifted 
over said forward motion vectors (D f ) to a temporal location of said image (II) to be 
interpolated, and by comparing said second forward estimation errors to said threshold. 

2. A method as claimed in claim 1, wherein said step of detecting said 

25 uncovered (I) and said' covered (III) parts in said neighboring previous (I) and next (III) input 
images includes the step of processing said backward estimation errors (e b ) and said forward 
estimation errors (e f ) to detect said uncovered (I) and said covered (III) parts in said 
neighboring previous (I) and next (III) input images, respectively. 
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3. A device for detecting covered and uncovered parts in an image (II) to be 
interpolated between neighboring previous (I) and next (III) input images, the device 
comprising: 

means for detecting motion from said next input image (III) to said 
5 previous input image (I), resulting in backward motion vectors (D 6 ) having corresponding 
backward estimation errors (e b )\ 

means for detecting motion from said previous input image (I) to said next . 
input image (III), resulting in forward motion vectors (D f ) having corresponding forward 
estimation errors (e f )\ 

10 means for detecting uncovered (I) and covered (III) parts in said 

neighboring previous (I) and next (III) input images, respectively; 

means for detecting in said thus detected uncovered parts (I), uncovered 

parts (la) in said image (II) to be interpolated by determining second backward estimation 

errors by comparing both neighboring previous (I) and next (III) input images when partially 
15 shifted over said backward motion vectors (D b ) to a temporal location of said image (II) to be 

interpolated, and by comparing said second backward estimation errors to a threshold; 

and detecting in said thus detected covered parts (III), covered parts (Illb) in 

said image (II) to be interpolated by determining second forward estimation errors by 

comparing both neighboring previous (I) and next (III) input images when partially shifted 
20 over said forward motion vectors (D f ) to a temporal location of said image (II) to be 

interpolated, and by comparing said second forward estimation errors to said threshold. 

4. A device for estimating motion vectors, comprising: 

a device for detecting covered and uncovered pans in an image to be 
25 interpolated as claimed in claim 9; and 

means for furnishing motion vectors for said covered and uncovered parts 
in said image (II) to be interpolated in dependence upon said backward motion vectors (D b ) 
and said forward motion vectors (D f ). 

30 5. A method of estimating motion vectors for an image (II) to be interpolated 

between neighboring previous (I) and next (III) input images, the method comprising the 
steps of: 

detecting motion from said next input image (III) to said previous input 
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image (I) t resulting in backward motion vectors (D b ) having corresponding backward 
estimation errors (e b ); 

detecting motion from said previous input image (I) to said next input 
image (III), resulting in forward motion vectors (D f ) having corresponding forward 
5 estimation errors (e f ); 

detecting uncovered (la) and covered (Hlb) parts in said image (II) to be 

interpolated; 

furnishing motion vectors for said uncovered parts (la) in dependence 
upon said forward motion vectors (D f ) taken from locations wholly and/or partly shifted over 
10 said backward motion vector (D b ); and 

furnishing motion vectors for said covered parts (Illb) in dependence upon 
said backward motion vectors (D b ) taken from locations wholly and/or partly shifted over 
said forward motion vectors (D f ). 

15 6. A method as claimed in claim 5, wherein said step of detecting said 

uncovered (la) and said covered (Hlb) pans in said image (II) to be interpolated includes the 
step of processing said backward estimation errors (e^ and said forward estimation errors (e ( ) 
to detect said uncovered (la) and said covered (Illb) parts in said image (II) to be 
interpolated. 

20 

7. A method as claimed in claim 5, wherein said steps of furnishing motion 

vectors for said uncovered (la) and covered (Hlb) parts comprise the steps of: 

furnishing first motion vectors for said uncovered parts (la) by taking said 
forward motion vectors (D f ) from locations wholly shifted over said backward motion vectors 
25 (D b ); 

furnishing second motion vectors for said uncovered parts (la) by taking 
said forward motion vectors (D f ) from locations partly shifted over said backward motion 
vectors (D b ) corresponding to a temporal location of said image (II) to be interpolated 
between said neighboring previous (I) and next (III) images; 
30 furnishing first motion vectors for said covered parts (Illb) by taking said 

backward motion vectors (D b ) from locations wholly shifted over said forward motion vectors 
(D f ); 

furnishing second motion vectors for said covered parts (Nib) by taking 
said backward motion vectors (D b ) from locations partly shifted over said forward motion 
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vectors (D f ) corresponding to a temporal location of said image (II) to be interpolated 
between said neighboring previous (I) and next (III) images. 

8'. A method of interpolating an image (II) between neighboring previous (I) 

5 and next (III) input images, the method comprising the steps of: 

estimating motion vectors for said image (II) to be interpolated in 
accordance with claim 5; and 

furnishing said image (II) to be interpolated by means of said backward 
motion vectors (D b ), said forward motion vectors (D f ), said motion vectors for said 
10 uncovered parts (la), and said motion vectors for said covered parts (Illb). 

9. A device for estimating motion vectors for an image (II) to be interpolated 
between neighboring previous (I) and next (III) input images, the device comprising: 

means for detecting motion from said next input image (III) to said 
15 previous input image (I), resulting in backward motion vectors (D b ) having corresponding 
backward estimation errors (e b ); 

means for detecting motion from said previous image (I) to said next 
image (III), resulting in forward motion vectors (D f ) having corresponding forward 
estimation errors (e f ); 

20 means for detecting uncovered (la) and covered (Illb) parts in said image 

(II) to be interpolated; 

means for furnishing motion vectors for said uncovered parts (la) in 

dependence upon said forward motion vectors (D f ) taken from locations wholly and/or partly 

shifted over said backward motion vector (D b ); and 
25 means for furnishing motion vectors for said covered parts (Illb) in 

dependence upon said backward motion vectors (D b ) taken from locations wholly and/or 

partly shifted over said forward motion vector (D f ). 

10. A display apparatus, comprising: 

30 a device (DE) for estimating motion vectors for an image (II) to be 

interpolated between neighboring previous (I) and next (III) input images as defined in claim 
9; 

means (MCI) for interpolating said image (II) between said neighboring 
previous (I) and next (III) input images by means of said backward motion vectors (D b ), said 
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forward motion vectors (D f ), said motion vectors for said uncovered parts, and said motion 
vectors for said covered parts; and 

means (CRT) for displaying an image sequence formed by said 
neighboring previous (I) and next (III) input images and said image (II). 





i 

FIG. 2 i 



■t 



} 



WO 99/22520 



PCT/IB98/01594 




PCT 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
H04N 7/26, 5/14, G06T 7/20 



A3 



(11) International Publication Number: 
(43) International Publication Date: 



WO 99/22520 

6 May 1999 (06.05.99) 



(21) International Application Number: PCT/IB98/01 594 

(22) International Filing Date: 12 October 1998 (12.10.98) 



(30) Priority Data: 

97202562.1 



29 October 1997 (29.10.97) 



EP 



(71) Applicant: KONINKLIJKE PHILIPS ELECTRONICS N.V. 
[NL/NL]; Groenewoudseweg 1, NI^5621 BA Eindhoven 
(NL). 

(71) Applicant (for SEonly): PHILIPS AB [SE/SE]; Kottbygatan 7, 

Kista, S-164 85 Stockholm (SE). 

(72) Inventors: DE HAAN, Gerard; Prof. Holstlaan 6, NL-5656 

AA Eindhoven (NL). PELAGOTTI, Anna; Prof. Holstlaan 
6, NL-5656 AA Eindhoven (NL). 

(74) Agent: STEENBEEK, Leonardus, J.; Intemationaal Octrooibu- 
reau B.V., P.O. Box 220, NI^5600 AE Eindhoven (NL). 



(81) Designated States: JP, European patent (AT, BE, CH, CY, DE, 
DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE). 



Published 

With international search report. 

(88) Date of publication of the international search report: 

5 August 1999 (05.08.99) 



D? 



(54) Title: MOTION VECTOR ESTIMATION AND DETECTION OF COVERED/UNCOVERED IMAGE PARTS 

(57) Abstract 

In a method of detecting 
covered and uncovered parts in an 
image to be interpolated between 
neighboring previous (I) and 
next (III) input images, backward 
motion vectors from the next 
input image to the previous input 
image, and having corresponding 
backward estimation errors, and 
forward motion vectors from 
the previous input image to the 
next input image, and having 
corresponding forward estimation 
errors, are determined, uncovered 
and covered parts are detected 
in the neighboring previous and 
next input images, respectively, in 
the thus detected uncovered parts, 
uncovered parts in the image to 
be interpolated are detected by 

determining second backward estimation errors by comparing both neighboring previous and next input images when partially shifted over 
the backward motion vectors to a temporal location of said image to be interpolated, and by comparing the second backward estimation 
errors to a threshold, and in the thus detected covered parts, covered parts in the image to be interpolated are detected by determining 
second forward estimation errors by comparing both neighboring previous and next input images when partially shifted over the forward 
motion vectors to a temporal location of the image to be interpolated, and by comparing the second forward estimation errors to the 
threshold. 



Db' 





















































































t, 
















































■ 




















i 








n 






























lb 












th 
























































































































/ 
























%- 




*>* 






* 


/ 















FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Armenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


-Burkina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


UZ 


Uzbekistan 


CF 


Centra] African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


ZW 


Zimbabwe 


CI 


COte d'lvoire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






cu 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







1 



INTERNATIONAL SEARCH REPORT 


International application No. 

PCT/IB 98/01594 


A. CLASSIFICATION OF SUBJECT MATI'liK 



IPC6: H04N 7/26, H04N 5/14, G06T 7/20 

According to International Palcnl Classification (IPC) or to both national classification an J 1 PC 

B. FIELDS SEARCHED ~ 
Minimum documentation searched (classification system followed by classification symbols) 

IPC6: H04N, G06T 

Documentation searched other lhan minimum documentation to the extern thai such documents arc included in the fields searched 

SE,DK,FI,N0 classes as above 

Electronic data base consulted during the international search (name of dala hasc and, where practicable, search terms used) 



C. DOCUMENTS CONSIDERED TO HE RELEVANT 



Category* 



Oration of' document, with indication, where appropriate, of the relevant passages 



Relevant lo claim No. 



GB 2279531 A (SONY UNITED KINGDOM LIMITED), 
4 January 1995 (04.01.95) 



US 5619272 A (PHILIPPE SALMON ET AL), 8 April 1997 
(08.04.97) 



EP 0778698 A2 (THOMSON MULTIMEDIA A.A.), 
11 June 1997 (11.06.97) 



GB 2265783 A (KENNETH STANLEY JONES), 
6 October 1993 (06.10.93) 



1-10 



1-10 



1-10 



1-10 



"xl J*urtlicr documents arc listed in the continuation of Ho.x C. See patent family annex. 



* Special categories of cilcd documents 

"A" document defining the general stale of the art which is not considered 

lo be or particular relevance 
*E" erlier document but published on or after the international tiling dale 
"L* document which may throw dnuhls on priority claim(s) or which is 

cued to establish the publication date of another citation or other 

special reason (as specified) 

'O* document referring to an oral disci osure, use, exhibition or other 
means 

"P* document published prior to the international filing dale hut later than 
the priority dale claimed 



*I " later document published after the international filing date or priority 
dale and not in conflict with the application hut cited to understand 
the principle or theory underlying the invention 

"X ' document of particular relevance: the claimed invention cannot be 
considered novel or cannot he considered lo involve an inventive 
step when the document it taker* alone 

" V" document of particular relevance the claimed invention cannot he 
considered ti> involve an inventive step when Ihc document is 
combined with one or more other such documents, such combination 
being obvious in a person skilled in the art 

&" document member of the same palcnl family 



Date of llic actual completion of the iritcrii.-ilioii.il search 
31 Mav 1999 


Date of mailing of ihc international search report 

0 2 -06- 1999 


Name and mailing address of the ISA- 

Swedish Patent Office 

Box 5055, S-102 42 STOCKHOLM 

Facsimile No. +46 R 66ft U2 80 


Autliori/.cd oflieer 

Michel Gascoin/mj 

Telephone No. + 4(» 8 7X2 25 DO 



2 

INTERNATIONAL SEARCH REPORT 



International application No. 

PCT/IB 98/01594 



C (Continuation). DOCUMEvNTS CONSIlJI-RliD 'I'D HI! Rlil.EiVANT 



Category* 


Citation of document, with indication, where appropriate, of (he relevant passages 


Relevant to claim No. 


A 


EP 0753972 A2 (PHILIPS PATENTVERWALTUNG GMBH) , 
15 January 1997 (15.01.97) 


1-10 



Form PCr/lSA/210 (continuation of second sheet) (July 1992) 



INTERNATIONAL SEARCH REPORT 

Information on patent family members 

03/05/99 


International application No. 

PCT/IB 98/01594 


Patent document 
cited in search report 


Publication 
date 


Patent family 
mcmhcr(s) 


Publication 
dale 



GB 2279531 A 04/01/95 JP 8009339 A 12/01/96 



US 5619272 A 08/04/97 DE 69321737 D 00/00/00 

EP 0677228 A,B 18/10/95 

FR 2700090 A,B 01/07/94 

JP 8505496 T 11/06/96 

SG 46694 A 20/02/98 

WO 9416522 A 21/07/94 



EP 0778698 A2 11/06/97 NONE 



GB 


2265783 


A 


06/10/93 


GB 
GB 


2265784 A,B 
2266638 A,B 


06/10/93 
03/11/93 


EP 


0753972 


A2 


15/01/97 


DE 
JP 
US 


19525531 A 
9084045 A 
5771074 A 


16/01/97 
28/03/97 
23/06/98 



Form PCX/1 SA/2 10 (patent family annex) (July 1992) 



